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PREFACE 



The Southern Regional Education Board (SREB) was awarded a grant (Me^^al 
Health Training Grant No. 1-T15-MH14703-01 ) in late 1976 from the Continu 
Education Branch of the National Institute of Mental Health (NIMH) to develop 
publications and conduct workshops to assist mental health centers in improv- 
ing their management practices and their program activities through the use 
of practical program evaluation. A series of publications and workshops are 
being developed through the combined efforts of SREB staff and task force 
participants. Topic areas include: 

§ The Administrative Uses of Program Evaluation 

• Practical Information Systems and Their Use in 
Monitoring Programs 

• Outcome and Effectiveness Studies 

• Assessing Needs and Setting Measurable Goals 

• Cost Analyses 

§ Utilization Review and Peer Review 

t Compliance with Standards of Federal, National and 
State Agencies and Organizations 

The project staff conducted a survey of mental health centers and clinics 
in the 14 states served by SREB to determine the kinds of program evaluation 
activities these centers and clinics were already doing and what their 
preferences were for workshops in program evaluation. These responses are 



being used to set priorities for project activities and develop a mailing list 
for publications and workshops. A strong interest in a workshop on the admin- 
istrative uses of program evaluation was expressed by the respondents. 

Using Program Evaluation in Mental Health Centers is designed to serve as 
an overview for a series of publications that will follow. Many of the ideas 
and suggestions expressed in this publication come from the task force merribers 
who met with us to assist in developing the content for this publication and 
the workshop. We thank them for their willingness to share their knowledge 
and experience in the everyday world of the center administrator and program 
evaluator as well as their expertise in management and evaluation methods. 
Because of their efforts, this publication represents a variety of perspec- 
tives on center administration and program evaluation. Staff, however, 
assumes responsibility for the content of this report including any mis- 
understandings resulting from translating the task force's ideas. 

Janet F. Despard, Project Director 
Improving Mental Health Centers and 
Mental Health Planning Project 

Harold L. McPheeters, M.D. 
Director, Commission on Mental 
Health and Human Services 
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Chapter I 

INTRODUCTION TO ADMINISTRATIVE USES OF PROGRAM EVALUATION 



The application of program evaluation activities in mental health centers 
has been limited. There are some remarkable sophisticated examples of certain 
kinds of program assessments in some centers, but generally the program assess- 
ment that is done, if done at all, tends to be used largely for formal report- 
ing to state and federal agencies or for preparing annual reports, news 
releases and the like rather than to be used for management decision making 
and day-to-day administration of programs. 

The Community Mental Health Centers Act of 1975 (PL 94-63) requires new 
or strengthened administrative and program structures in community mental 
health centers. Under this law, centers are expected to implement program 
evaluation and related activities in three areas: 1) quality assurance 
procedures; 2) self evaluation; and 3) programmatic compliance with federal 
requirements. Centers are also required to provide two percent of their 
previous year's total operating budget for financing program evaluation 
activities. 

In response to this legislation, many centers, especially those receiv- 
ing grants under the Community Mental Health Centers Act of 1975, are work- 
ing toward developing appropriate program evaluation capabilities. Ideally, 
program evaluation activities include assessing needs for services, setting 



program objectives with measurable outcome criteria, information systems, 
data monitoring, utilization review, peer review, standards monitoring, out- 
come studies (including consultation and education services), cost analyses, 
client satisfaction studies, and special studies. Very few mental health cen- 
ters, however, have been able to actually implement this full range of program 
evaluation activities. A few of the larger and better financed centers can 
expect to achieve a high level cf proficiency and sophistication in some of 
these activities, but it is doubtful that the average center with a total 
annual budget of about a million dollars can ever plan to have such elaborate 
program evaluation systems because of limited funds, limited expertise and 
limited access to technical support services. Furthermore, federally funded 
mental health centers are only part of the total mental health center picture. 
There are many mental health centers that do not receive direct grants under 
the Community Mental Health Centers Act but are partially funded by and 
operate under the regulations set by state mental health authorities. 
Although considerable variation exists from state to state, these centers 
must also provide evaluative information on program activities and the use 
of resources to their state mental health authorities and must comply with 
state requirements for mental health programs. Many of these agencies also 
have a need and a desire to enhance their program evaluation activities. 

Several trends have made program evaluation in centers a growing neces- 
sity. General economic conditions have reduced funds available for mental 
health programs in many states. Grants awarded under the Community Mental 
Health Centers Act of 1975 require zero-based annual budgets and mandate 
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that centers receiving new funding make every possible effort to increase 
the amount of client fees and third party reimbursements collected. Addi- 
tionally, since 1975 more and more centers funded under previous community 
mental health legislation (PL 89-105) are reaching the end of their staffing 
grants and are faced with the need to expand and diversify their base of 
funding. Often, these agencies must turn to their state mental health 
authority and local government for additional funding to meet the demand for 
services. 

Some centers are now seeking accreditation by the Joint Commission on 
the Accreditation in Hospitals (JCAH) to increase third party reimbursements. 
But to meet the standards set by this organization, centers often must 
upgrade their existing clinical and management procedures, particularly those 
related to program planning, case management, quality review and program 
evaluation. 

While the economics of center operations become more critical, the new 
federal amendments broaden the range of services required of mental health 
centers to include drug and alcohol treatment and rehabilitation, special 
services for the elderly and children, screening, follow-up and transitional 
care. Many states have followed this general model of merging mental health, 
drug and alcohol treatment and rehabilitation services and, in some, states, 
mental retardation. The management skills and ingenuity of a center are 
taxed under these conditions. Evaluation information can prove to be useful 
in assisting administrators to make appropriate decisions in order to provide 
services within budgetary limitations. 
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The new legislation also requires that centers begin quality assurance 
procedures. (As Professional Standards Review Organizations become estab- 
listed, many centers will be faced with external quality assessments of their 
inpatient services as well.) Centers are expected to be accountable to the 
communities that they serve by encouraging residents of the catchment area 
to review center operations as depicted by evaluation findings. Centers are 
also expected to be responsive to citizens' suggestions for improving services. 

Finally, the Health Planning and Resources Development Act of 1974 
(PL 93-641) which created a new system of Health Systems Agencies on the 
local level and Statewide Health Coordinating Councils on the state level — 
will influence mental health services in the coming years. Although the 
impact is not yet evident, this legislation incorporates mental health plan- 
ning and program development under the rubric of comprehensive health, 
requiring that Health Systems Agencies approve or disapprove many mental 
health, drug abuse, and alcoholism project grants provided by the federal 
government. 

Overall these trends indicate that mental health centers are faced with 
the challenge of insuring their survival by improving ?.id strengthening their 
administrative and program evaluation capabilities. They must be able to 
provide a broad range of quality services to the community, actively pursue 
a more diversified funding base, communicate better internally and with their 
environment, and encourage the support, the participation and the constructive 
criticism of community groups, other human service agencies and local govern- 
ment. Practical program evaluation activities can produce the information 
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that will assist center administrators and others in making the decisions and 
taking the actions necessary to meet this new challenge. 

PURPOSE 

The purpose of this publication is to provide an overview of the ways 
that practical program evaluation activities can assist centers in improving 
their administrative and program capabilities. 

In this introductory chapter, the terms and assumptions used in the 
publication are defined, and an outline of the stages of the program evalua- 
tion cycle is presented. 

The second chapter describes the program evaluation activities that are 
a part of the outline and the roles of the administrator and the evaluator 
in conducting evaluation activities. It is suggested that the reader refer 
to the subsequent publications produced by this project and other references 
for more detailed descriptions of various evaluation methodologies and 
instruments. 

The third chapter presents alternative ways to organize and deploy 
staff for program evaluation activities. It is our position that management 
functions should support and facilitate the delivery of mental health serv- 
ices and that program evaluation is a part of these management functions, 
not primarily a research activity. Because of limited resources, small 
centers and clinics may not be able to employ a full-time program evaluator 
but must assign evaluation responsibilities to a multi-purpose unit or to 
staff members who carry other duties as well. The desirable skills and 



training of an evaluator include a broad range of technical and interpersonal 
skills seldom possessed by one individual. Therefore, the actual selection 
of an evaluator depends largely on the priorities set by a center's manage- 
ment and the skills of the existing staff. The e;aluator should be a member 
of the center's management team who maintains good working relationships 
with clinical and administrative staff and has the flexibility to be able to 
perform a wide variety of tasks. 

In the final chapter, the importance of the leadership skills of the 
center's administrator in creating a positive climate for program evaluation 
is stressed. One of the most significant issues involved in beginning pro- 
gram evaluation activities is seeking a balance between the need to improve 
management procedures and thj need to maintain or improve the quality of 
care provided to clients. Program monitoring assists in creating this bal- 
ance by detecting variations in program activities that suggest inunediate 
action or further study. Evaluation findings drawn from monitoring reports 
and special studies provide useful information for planning center activities 
and assisting governing boards in making policy decisions. The review of 
evaluation findings by community groups encourages feedback regarding their 
suggestions for modifying the mental health services and may prompt indi- 
viduals and agencies in the community to offer resources that will aid the 
center in meeting these suggestions. 




DEFINITIONS 

Many of these definitions were adapted from the SREB publication. 
Definition of Terms in Mental Health, Alcohol Abuse, and Mental Retardation^ 



Program Evaluation 

A type of applied research in which program process and outcome- 
characteristics are related explicitly to a set of values, such 
as goals, objectives and costs. ^ in addition, evaluation 
identifies the problems and the side effects present in program 
activities. 

Program Evaluation is Based on Comparison 
Program evaluation compares actual operations against a norm, 
standard or criterion. These standards of comparison vary 
depending on the values of the person doing the evaluating and 
the anticipated uses of the findings. A center's management 
team sometimes sets different criteria for making judgments 
about a program than those set by funding agencies or com- 
munity groups. Therefore, it is important to be clear and 
explicit about the values and criteria being used. 

Working Definition of the Program Evaluation Process 
A systematic set of data collection and analysis activities 
undertaken to determine the value of a program. This infor- 
mation is used to aid management, program planning, staff 
training, public accountability and promotion.^ 



Program evaluation is a management subspecialty. Its purpose 
is to provide administrators with information to 1) determine 
at periodic intervals whether the activities and the use of 
resources are reaching established objectives; and 2) make 
changes in activities or the use of resources based on reason 
able judgments about efforts, effectiveness, efficiency, 
appropriateness and comparative value of options to assure 
that objectives are met at reasonable costs. 

Cyclical Stages of Program Evaluation 

Program evaluation is a cyclical process that involves three 
stages: 

• The PLANNING STAGE includes setting objectives for 
program activities and resources which define expec- 
tations for performance and criteria for measuring 
outcomes. This planning stage is an integral part 
of the development of activities and resources by a 
center's management team to meet the needs of the 
community for mental health services. 

• The MONITORING STAGE involves the systematic collec- 
tion, tabulation and analysis of data in periodic 
reports. The center administrator monitors these 
reports to determine whether any changes in activ- 
ities or reassignment of personnel or budget appear 
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PROGRAM EVALUATION CYCLE 



This Simplified model was developed as a general framework to demonstrate the three stages of program evalua- 
tion. It was based on reconmendations made by task force members and concepts found in other publications. 



HOOIFY PROGRAM 
ACTIVITIES AND/ORU 
USE OF RESOURCES 



PLANNING STAGE 
SET GOALS AND OBJECTIVES 
FOR PROGRAM ACTIVITIES AND 
RESOURCES BASED ON PREVIOUS 
EVALUATION FINDINGS AND 
ASSESSMENT OF NEEDS 




MONITORING STAGE 
COLLECT, lABULATE AND REPORT 
PERIODICALLY TO CENTER MANAGE- 
MENT ON PROGRAM ACTIVITY AND 
USE OF RESOURCES 




CONDUCT SPECIAL 
^STUDIES OF PROGRAM 
^ ACTIVITIES AND/OR 

USE OF RESOURCES 



CONTINUED MONITORING 
COLLECT, TABULATE AND REPORT 
PERIODICALLY TO CENTER MANAGE- 
MENT ON PROGRAM ACTIVITY AND 
USE OF RESOURCES 



ASSESSMENT STAGE 
ANALYZE DATA AND INTERPRET 
FINDINGS ON ATTAINMENT OF 
GOALS AND OBJECTIVES, OUTCOMES, 
IMPACT, SATISFACTION, COSTS. 
CONDUCT SPECIAL STUDIES IF 
NEEDED. REVIEW OF FINDINGS BY 
MANAGEMENT, GOVERNING BOARD, 
FUNDERS AND COMMUNITY GROUPS. 



FEEDBACK AND 
TeTCTm E N dXT Ilf fST 



SOURCES: Hargreaves, W. A., Attkisson, C. C. , and Sorensen, J. E., Resource Materials for Community Mental 
Health Program Evaluation (2nd Edition). DHEW Publication No. ADM 77-328, 1977; Franklin, Jack L. 
and Thrasher, Jean^ An Introduction to Program Evaluation . New York: John Wiley and Sons, Inc., 
1976; and Little, Arthur D., Inc., A Working Manual of Simple Program Evaluation Techniques for 
Community Mental Health Centers . DHEW Publication No. ADM 76-404, 1976. 
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necessary to assure that objectives are met. These 
reports are used like a thermostat that shows when 
activities do not meet an accepted or expected range 
and changes are needed. 

• The ASSESSMENT STAGE consists of analyzing and inter- 
preting the evaluation reports and other data to deter- 
mine the degree to which programs have met the criteria 
set in the original objectives. It also defines the 
maroower effort and costs involved in meeting these 
objectives. Other routine evaluation activities may 
include studies of client outcomes and impact, cost 
analyses, and client satisfaction. The findings of 
the assessment stage are prepared for review by the 
center's management team, governing board, funding 
agencies and community groups. These findings and 
community recommendations are then used as the base 
for the next planning stage. 

Special Study 

A special study involves the collection and analysis of data that 
are not routinely planned as part of the periodic reporting process. 
Data for special studies may be drawn from existing information 
sources or may require separate data collection procedures. 
Centers should have the capabilities for conducting selected 
special studies for use in the decision making process. Variations 
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in regularly monitored data often call attention to the need for 
special studies, or special studies may be initiated by curious 
staff or upon requests from board members or outside groups. The 
nature of the study may be very simple (e.g., a "no-show" study) 
or relatively complex (e.g., a comparative study of treatment out- 
comes). In the context of this publication, any evaluation 
approach that is not routinely planned is a special study. There- 
fore, the definition of a special study depends largely on the 
kinds of routinely scheduled program evaluation activities going 
on in an individual center. For example, one center may routinely 
collect and analyze data related to client satisfaction in which 
case this activity is not a special study. Another center may 
undertake this activity only when prompted by complaints from 
local groups or clients. This activity is then considered a 
special study. 

Evaluative Research 

Evaluative research is a particular kind of evaluation which 
involves the application of scientific methods to management 
decision making about the overall worth or productivity of a 
program.^ These studies differ from program evaluation in that 
they require higher validity, rely more on experimental research 
designs and are time-limited, not ongoing. Evaluative research 
may involve discovering of new knowledge or testing the appli- 
cation of such new knowledge in an organized program. Many 
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mental health centers do not have the staff expertise or the 
funds to conduct evaluative research. It is recommended that 
centers implement and maintain a well developed program evalua- 
tion system before getting involved in evaluative research studies 
which have limited utility in resolving immediate program manage- 
ment problems. The findings resulting from evaluative research 
are useful, however, in long-term planning for program develop- 
ment and change. 

Mental Health Program 

A set of related organizations, resources and/or program trans- 
actions directed to accomplish a defined set of objectives for 
a specific target population in a specific geographic area. 

It is possible that "program" may be synonymous with "organization" 
when only one organization exists or with "service" when only one 
service is offered. Examples include alcohol treatment program, 
mental retardation program, and children's program. Programs may 
be classified in a variety of ways based on administrative ties, 
staff services, recipients or purposes. 

Center Administrator 

A person who has responsibility for directing the program or 
programs and managing the resources of a community mental health 
center. 
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Management Team 

A multi-disciplinary group of staff members who coordinate their 
skills to provide the management direction to the organization. 
In the context of this publication, the management team includes 
the center administrator, the business and/or fiscal officer, 
the clinical program directors, and the staff memcters who are 
responsible for personnel, staff training, public information, 
data processing statistics, program evaluation and program 
planning. 
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Chapter II 
PROGRAM EVALUATION ACTIVITIES 



Many program evaluation activities in an average mental health center 
are designed primarily to meet federal and state requirements for reporting 
of program activity data. These same reporting activities also yield infor- 
mation that is useful for planning and managing of center programs. Because 
there is a broad range of interrelated activities included in each of the 
three stages of program evaluation outlined in Chapter I, only brief general 
descriptions of the activities that fit into each of these stages are given 
next. These descriptions are based on those found in an earlier SREB 
publication. Program Evaluation in the State Mental Health Agency , unless 
otherwise noted. 

PLANNING STAGE 

A center administrator is responsible for planning, at least on an 
annual basis, for the services to be provided; developing an annual budget 
for these services in cooperation with fiscal management; setting goals and 
objectives for the center and its programs; and developing proposals for 
funding of ongoing or new programs. The evaluator's role in the planning 
process will vary according to the organizational structure, the skills of 
the staff, the priorities and values of the center's administrator, and 
staff time available. Although not primarily responsible for most of these 
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planning decisions, the program evaluator should be actively involved. The 
evaluator will at least be called on to provide data and should participate 
in the planning and decision making sessions so that he or she will have a 
clear understanding of the philosophical direction of the programs to be 
evaluated. 

Needs Assessment 

The first stage of the planning process — which is usually done when a 
center is just being started or when it is required to add new program com- 
ponents to ongoing operations — is assessing the target population's needs 
and expectations for services. Needs assessment, done before the development 
or implementation of programs, focuses on questions about the prevalence of 
mental health problems, the total number of people who need services, the 
demographic and special social characteristics of the population in need, 
projections of the probable demand for services, and information about exist- 
ing programs in the area which already meet part of this need. When planning 
for a program, the management team must decide on program priorities, the 
appropriate location of services, resources available (e.g., staff, space 
and funds) and special problems related to the delivery of services. The 
evaluator often does much of the assessment of needs because he may be the 
only person in the organization with the data gathering and analyzing skills 
required. 

t Many kinds of needs assessment methods have been developed and 
are available for use. Some of these methods are very expen- 
sive (e.g., sample surveys) while certain others, although they 
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may not produce as scientifically accurate results, may be just 
as satisfactory if they adequately support a visible need for 
services and can be used for measuring the impact of services 
at a later date (e.g., rates of people under treatment, infor- 
mation from key informants, community forum approaches, use of 
social indicators). 

§ Community mental health administrators should explore sources 
of existing data before collecting new data. Some potential 
existing data sources are city and regional planning bodies, 
health departments, mental health associations, health systems 
agencies, universities, state and federal funding agencies 
and national information clearinghouses.l 

• The program evaluator can be helpful in identifying useful 
existing data sources and selecting appropriate methods for 
needs assessment. The evaluator should also be included 
throughout the program planning and development process so 
that he or she will be familiar with the underlying social, 
philosophical and political realities which affect program 
development and subsequent program evaluations. 

The expectations of community leaders and groups should be examined in 
the needs assessment process. Regardless of how scientific and accurate a 
study of the need for services may be, the expectations of clients, community 
groups, professional associations, and local government have an influence on 
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the potential demand for services. Information drawn from community leaders, 
community forums, cultural analysis, and the nominal group approach are 
useful in identifying what the community feels it wants. 

Goal Setting 

An important part of planning and managing programs, whether newly ini- 
tiated or ongoing, is defining the organization's treatment philosophy and 
setting goals and objectives . In many centers, representatives from manage- 
ment and professional staff participate in annual review of the attainment of 
the previous year's goals and objectives and the development of new statements 
for the coming year . This process serves dual purposes by providing the eval- 
uator with criteria for measuring program outcomes and giving staff a clearer 
understanding of the expectations for their performance, 

A written statement of the center's philosophy and purposes should be 
part of the annual goals and objectives so that the values which govern the 
delivery of service within a program will be explicit and understood by all 
involved. 

If the center's philosophy is not made explicit by management, diffi- 
culties are likely to arise in programs. For example, if management assumes 
that program operations should provide basically adequate care to as many 
clients as possible while the therapists working within the program feel that 
they should provide the highest quality care to only a few clients, an eval- 
uation of outcome is likely to place management in an adversary position 
with the therapists in the program. 
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Some of the major issues that should be made explicit in a center's 
statement of philosophy are whether the goals of treatment are improved 
social functioning or removal of psychopathology; whether the dignity and 
freedom of clients are more important than the standardization and regimenta- 
tion of client care; and whether the program is committed or opposed to par- 
ticular models of care (e.g., medical, behavioral, or social) or procedures 
of treatment (e.g., psychoanalysis, psychopharmacology, group therapy, etc.). 

After defining the center's philosophy, the administrator, the- evaluator, 
other center staff and community advisory groups set goals and objectives for 
the coming year . Goals are concise descriptions of the desired end-states 
related to human needs to be sought at some future time. Goals are often 
expressed for a mental health center as a whole. Objectives are often used 
for specific programs or services within a mental health center. (There is 
some disagreement on the terminology used for this process. Some prefer to 
reverse the meanings of these terms.) There are five basic criteria for both 
goal and objective statements: 1) set a deadline for the activity; 2) name 
the target group; 3) state the desired outcome or end-state; 4) set the con- 
ditions or restrictions under which the desired outcome may be expected to 
occur; and 5) specify the minimum criteria for measuring the outcome desired 
as evidence that the objective was reached. 2 

All too often an evaluator is expected to evaluate ongoing programs 
without criteria for measuring outcomes. It is recommended that each center 
set objectives for programs as part of an annual planning cycle to provide 
specific criteria and make clear the expectations for program directions 
and staff performance. 
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Community mental health center administrators and their staff period- 
ically prepare proposals to obtain funds for ongoing or new programs. The 
process of assessing needs for services, designing programs to provide these 
services, and setting objectives and budget for programs are all part of 
writing proposals. If they have been engaged in a regular annual planning 
cycle, the administrator and evaluator will have available much of the infor- 
mation needed for proposal writing. 

MONITORING STAGE 

Few mental health centers have been able to develop and put into oper- 
ation a full range of program evaluation activities. Most mental health 
organizations have some system, however, for measuring effort -- who does 
what for whom at what cost - and for defining the basic demographic char- 
acteristics of the clients who receive services. These data are generally 
required for formal reports to state and federal agencies and to other 
funders. The scope and accuracy of these data are largely dependent on the 
adequacy of the data collection procedures including whether standard 
definitions of terms have been used and whether there has been full report- 
ing of the data. 

Information Systems 

The first stage in developing program evaluation capability is to organ- 
ize systematic ways of collecting and tabulating data about the work involved 
in providing services and the services themselves, commonly referred to as an 
" information system ." (Much of the needed data are presently available in 
most mental health centers, but they may not be well organized or convenient 
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to use.) Frequently, the term "information system" is interpreted to mean a 
sophisticated, computerized system for collecting and tabulating data. We do 
not use the term in this context. Instead, our usage refers to "a system for 
gathering information, accumulating that data in an organized file, and sum- 
marizing that information in periodic reports or in reports responsive to 
special requests. Such a system may be an extensive, automated one or it 
may be a relatively simple system of files and records with the capability 
for yielding basic information on client movement, staff activity and cost of 
services. These data show the quantity and overall cost of what a center is 
doing and can be used for internal monitoring of programs, to detect overuse 
or underuse of resources (services, manpower, facilities, funds), to prepare 
reports to funders and to provide baseline data for many planning, evaluation 
and utilization review activities. 

Monitoring 

In addition to satisfying other data needs, information systems can be 
used for monitorin g j)rograms_. It is important that the data and reports be 
kept up to date so that the center administrator is aware of significant 
changes in operations and can make changes in these programs when indicated. 
A staff member should be assigned the responsibility for monitoring data and 
compiling periodic reports for the center administrator regarding client 
movement, staff activity and costs. The frequency of these reports will 
vary. The basic rule of thumb is to prepare reports at regular intervals 
that are far anough apart to reflect trends. Monthly or quarterly reports 
in most areas are adequate. The program evaluator can then examine these 
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reports to determine whether ongoing activities measure up to expectations. 
If they do not, the evaluator should bring any variations to the attention of 
the administrator for a decision on whether further inquiry into ongoing 
operations is necessary. 

Another kind of monitoring is related to the use of standards (a "state 
or condition accepted as a minimal or exemplary condition, appearing in law, 
regulation or policy").* Many centers, as well as state, federal and national 
organizations have developed standards or are now formulating them in response 
to the requirements of third party payment programs, policy and legislation 
regarding cost containment and quality assurance. 

There are generally three types of standards: 

t Input standards are those that spell out the basic resources 
required for programs. These include such items as building 
standards, staffing ratios, staff qualifications regarding 
training, licensure and certification, and equipment standards. 

• Process standards are those which define the procedures to be 
used in the clinical and administrative services. 

• Outcome standards are those that spell out the client outcomes 
to be attained. Outcome standards would be most ideal from 
the perspective of the program evaluator, but they are rarely 
used because it is difficult to develop outcome standards in 
human service work. 
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Outside agencies may monitor program activities within a center through 
site visits or through reports submitted to them by the center. However, the 
responsibility for complying with established standards and procedures rests 
with the center itself. The evaluator can play an important role in helping 
the administrator monitor standards, deciding how to measure standards and 
establishing procedures to detect problems that need to be brought to the 
attention of the administrator for modification. 

Quality assurance procedures require monitoring and review of clinical 
services. Both utilization review and peer review are included in these 
procedures. 

Utilization review monitors and evaluates the appropriateness and use of 
a program's services. It is used to assure that clients are not under treat- 
ment unnecessarily or kept under treatment longer than necessary. Utiliza- 
tion standards are set and reviewed by special committees, but the program 
evaluator can help in setting these standards and periodically reviewing the 
results of the review committee's actions to determine whether there are 
overall problems which require corrective action. Analysis by the evaluator 
might also determine that the standards used by the utilization review team 
need modification. 

Peer review is a mechanism for evaluating treatment and rehabilitation 
procedures by a formal review of clinical records by a team of peers. Until 
recently peer review was done when a complaint was filed as a result of an 
excessive fee or a poor outcome. Now, with the new federal legislation 
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(PL 94-63), the increase in third party programs and Professional Standards 
Review Organizations (PSROs) many centers must set and monitor standards 
for all treatment cases. Under PSRO, inpatient care will be monitored and 
reviewed according to the clinical criteria set by peer practitioners in a 
specific facility or by an organization of peers in the community (a PSRO). 
In either case the evaluator might be involved in helping set the original 
criteria and periodically examining the peer review actions of the program's 
treatment services to detect problems or trends in both the program and the 
peer review process that should be brought to the attention of the 
administrator. 

ASSESSMENT STAGE 

Periodically the results of ongoing programs are reviewed and analyzed 
using some of the following program evaluation methods. The frequency of 
these evaluations depends on the center's capabilities, the schedule for 
required reports of program activity and how often monitoring reflects prob- 
lems which require special studies of a program. 

Outcome Studies 

Outcome studies show how well and to what extent programs have met their 
objectives and whether there are any unexpected outcomes or spin-off effects. 
Generally, the center administrator, the evaluator and others set program 
objectives, using baseline data, norms, and standards as criteria to measure 
the outcomes. 

A wide range of individual client and program outcome measures are avail- 
able for use by the evaluator. There are two basic types of measures: 
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standardized instruments which measure changes in psychopathology, symptoms 
or general functioning of clients, and goal attainment scaling techniques 
which measure specific individualized goals of clients. Both require train- 
ing for administration and analysis. With few exceptions they also require 
both pre- and post-measurement. 

Impact Studies 

Impact studies a nalyze the relationships of the program outcomes to the 
original n eed and to any related consequences . Related consequences may be 
economic, social, political or clinical. Because they go beyond immediate 
outcomes, impact studies are one of the more comprehensive evaluation tech- 
niques for letting the administrator know the total effect of a program. 

Impact studies are difficult to design, implement and interpret. Most 
impact studies done by centers are tied to an early assessment of need for 
services, using pre- and post-social indicators or sample surveys. Theoreti- 
cally, impact studies imply a causal relationship between an agency's activ- 
ities and the social well-being of a community. Methodologically there are 
numerous design problems, chief of which is the intrusion of factors beyond 
the control of the agency involved in providing services. Many societal and 
cultural forces in the community work against the scientific evaluation of 
human services programs (e.g., mobility, migration, economic fluctuations, 
changes in national policy, demographic changes within the community, altered 
physical or ecological patterns). 

Some centers now measure the impact of services on clients by focusing 
on short-term effects of treatment by using pre- and post-measures of 
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social functioning. Although this approach does not eliminate intrusive 
factors, it does narrow the population under study to actual clients, thus 
controlling some of the external influences that make impact difficult to 
measure. 

Cost Analysis Studies 

Cost analysis studies analyze program expenditures according to various 
criteria . Essentially, costs are figures derived by allocating expenditures 
according to some significant measure (e.g., cost per patient per day, cost 
per episode of care, cost per patient year). These costs can then be ana- 
lyzed and comparisons made between similar programs, between successive 
years, between different models of treatment, and between different patterns 
of organization. Cost studies of this kind are valuable to the administrator 
as predictors for setting priorities, making program changes and preparing 
budgets. 

It has been common for cost studies, when they have been done at all, to 
be done by the business office with very little relationship to the clinical 
programs. The studies have thus tended to be concentrated in the business 
area only (e.g., costs of food, supplies or personnel rather than costs of 
treatment, aftercare, rehabilitation or crisis services). 

Cost outcome studies measure the cost of providing services to groups of 
clients in relation to the degree of improvement in their functioning over 
time. This kind of study can be used to discover the factors that contribute 
to differences in costs and client outcome and to provide baseline data for 
decisions about the best ways to deliver services to specific client groups. ^ 
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Cost-benefit analysis compares the cost of a particular effort with bene- 
fits obtained from it . It attempts to assign monetary value to benefits and 
then divides this figure by the cost in dollars. Presently, determining bene- 
fits and assigning monetary values to them is r^^uch more an art than a science 
because it is very difficult to describe social benefits in common monetary 
units. Economists readily agree that the techniques for assigning monetary 
values to social benefits are not yet firmly defined. Cost-benefit analysis 
is generally used now as a projective tool to assess the relative effective- 
ness of proposed program alternatives. Although economists may disagree, 
and the technique is basically projective, it may still be useful to the 
evaluator and administrator in assessing present and past benefits of a 
program, 

Cost-effectiveness studies are a limited version of the cost-benefit 
technique which attempts to specify and evaluate social costs and benefits of 
different programs and services that have the same target population and 
identical predefined goals. Since the target problems are the same, whatever 
measurement of benefit is applied to one group is applicable to other groups. 
For example, if one group of neurotic depressives received psychotherapy 
alone and another group of neurotic depressives received drug therapy alone, 
the measures of treatment relevant for one group -- decreased depression and 
improved family relationships ~ are as relevant for the other group. There- 
fore, cost-effectiveness studies are less vague than the more general cost- 
benefit data and allow the administrator to feel more confident when judging 
alternatives. These studies are a form of evaluation research rather than 
everyday program evaluation. 
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Client Satisfaction Studies 

Client satisfaction studies analyze the opinions, attitudes and reac- 
tions of clients about the services they receive from a program . These 
studies indicate how the program is meeting the expectations and needs of 
the clients, their families or referral agencies. There are various tech- 
niques including questionnaires, personal visits and telephone interviews, 
to evaluate whether these people feel they have been well served, what prob- 
lems or shortcomings they have experienced and what suggestions they may 
have for improving the services. The evaluator should be alert to specific 
suggestions made by individuals on client satisfaction studies, for these 
suggestions may provide leads for significant improvements in programs. 

Special Studies 

Special studies examine are as where there are no routine data collec- 
• :ion and analysis activities or where routine analysis indicates a need for 
further study. Also, it could develop that certain studies are needed only 
episodically (e.g., studies of staff time commitments to specific activities). 
Other studies are required only one time (e.g., to supply information for an 
investigation exploring charges that have been made against some specific 
aspect of the program). Special studies may be done on managerial functions 
(e.g., costs of central purchasing compared to decentralized purchasing) or 
on clinical problems and programs (e.g., analysis of a rising suicide 
attempt rate or of an increasing rate of seclusion and restraint in certain 
program units) . 
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The need for doing a special study should be prompted ordinarily by the 
monitoring activity of the program evaluator who senses a need or problem 
that requires special study. However, special studies may also be prompted 
by curious individuals within the agency's staff, by the administrator him- 
self, or occasionally by charges brought by the press, citizen groups or 
clients which require a special study as part of an investigation. 
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Chapter III 
ORGANIZATION AND STAFFING 



The scope of program evaluation activities and the assignment of program 
evaluation tasks in a center are influenced by a number of factors including 
the existing organization, supervisory and staffing patterns, the center's 
philosophy, the attitudes of management and staff toward program evaluation, 
perceptions of what program evaluation is, available resources, and the pri- 
orities of management. It is the position of this document that management 
functions should facilitate and support the delivery of direct services; 
program evaluation is not primarily a research activity, but rather a part 
of the management function. 

Small centers and clinics with limited financial resources and staff 
are often not able to hire a separate staff person with program evaluation 
expertise but must rely on the skills of existing staff to do minimal, 
practical program evaluation. In many cases, these organizations need the 
assistance of consultants in selecting appropriate program evaluation pro- 
cedures and training staff in the use and interpretation of data. Larger 
centers with greater financial resources and more supervisory staff must 
decide on the best ways to use the skills of an evaluator and to introduce 
this new staff member into the organization in a way that minimizes the 
resistance of others. 

« 

31 



37 



Because they are primarily clinicians without knowledge of the potential 
uses of program evaluation as a management tool, many center administrators 
have difficulties in recognizing the many ways that a program evaluator can 
be helpful as part of the management team. Sometimes they confuse program 
evaluation and its related activities with evaluative research. Program 
evaluation is an integrated system of collecting and monitoring data, and 
preparing timely reports on findings for decision making by management; on 
the other hand, evaluative research deals with research on specific program 
technologies and only secondarily with determining the effectiveness of on- 
going programs or services for management purposes. While evaluative research 
has almost no utility in resolving immediate management problems that surface 
every day in a center, program evaluation can be a valuable day-by-day man- 
agement tool if used appropriately. Center administrators are sometimes 
reluctant to take funds away from direct services for program evaluation 
activities. But, if used to aid in sharpening center operations, practical 
program evaluation should p ' for itself by assisting management in monitor- 
ing center activities and making decisions regarding allocations and program- 
matic changes that increase the efficiency and effectiveness of services. 

ORGANIZATION 

There is considerable variation in the ways that the organization and 
staffing of program evaluation can be structured within a mental health 
center. It is impossible to outline a single ideal structure because of 
the many factors which influence how program evaluation may be done within 
an organization. One generalization can be made, however: the center 
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administrator is responsible for the internal operations of the organization 
and should define the organizational placement, staffing and responsibilities 
for program evaluation. 

There are four basic alternatives for the organization and staffing of 
program evaluation: 

• Program evaluation as a separate unit; 

c Program evaluation combined with other administr' .ive or 
clinical services; 

• Assignment of program evaluation tasks to varied staff with 
coordination by the center administrator; 

t Program evaluation conducted by outside consultants. 

Program Evaluation as a Separate Unit 

A center may choose to establish a separate unit for program evaluation , 
staffed by a director who reports directly to the center administrator. This 
arrangement is feasible when the attitudes toward evaluation within the center 
are positive and the need can justify funding a separate unit for this activ- 
ity. The administrator and the evaluator should have a close working relation- 
ship and share the skills needed to use evaluation as a managei^r^nt tool as 
well as to prepare the reports required by funding agencies. If center admin- 
istrators are not skilled in using program evaluation as a msr.uqemcnt tool, 
they may not recognize the evaluators' potential contr-^iutions to internal 
management* In addition, if evaluators are unfamiliar >h the use of eval- 
uation for internal management, they may be incVned to want to do evaluative 
research which is not likely to be as relevant, timely or useful to manage- 
ment as practical program evaluation. Also the attitudes of oth^r staff may 
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influence the effectiveness of the evaluator. Sometimes the evaluator is 
seen as a threat by staff, particularly when the evaluator works in relative 
isolation. Clinical program staff should be included in the evaluation pro- 
cess and should be able to request technical assistance and data from the 
evaluator when they need information about their services and when they are 
planning changes in ongoing services. 

The evaluator in this structure can expect to be involved in a wide 
range of activities that are not always perceived as program evaluation. In 
all probability, he will be called upon to assist in the overall planning for 
the center, to monitor and report to the director and the management team on 
various center activities, to assist in quality assurance procedures, to set 
up data collection systems, to be involved in program development and change, 
and, in rare cases, to assume some of the administrative responsibilities of 
the center director, especially in preparing proposals for funding and 
reports to funding bodies, boards and community support groups. 

Program Evaluation Services 
Comoined with Other Sup p ort Services 

Many center administrators assign joint responsibilities to a unit within 
the organization . The possible combinations vary according to the needs of a 
center. For example, some centers combine evaluation and training or eval- 
uation and research; others combine planning and evaluation; others assign 
program evaluation to a clinician who has knowledge of evaluation methodol- 
ogy; others place evaluation with fiscal management, clinical records, or an 
administrative assistant to the center director. Whatever the combination of 
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responsibility may be, it is recommended that the person assigned respon- 
sibility for program evaluation work closely with the center administrator 
in defining what kind of program evaluation is needed and have the necessary 
skills or technical assistance available. A possible problem with the com- 
bined unit model is that the demands (or preferences) for other activities 
may have priority over program evaluation with the result that little eval- 
uation is done beyond the minimal required reporting. It is preferable that 
the functions combined with program evaluation be in close collaboration with 
administrative support services as well as with clinical services. 

Dispersed Program Evaluation 

Many center administrators prefer to assume primary responsibility for 
the coordination of program evaluation activities, assigning tasks to manage- 
ment and clinical staff according to their individual skills and ongoing 
responsibilities. This arrangement is feasible when administrators have work- 
ing knowledge of all of the center's operations and are familiar with eval- 
uation methods so that they can define the kind of data that are needed and 
the methods that should be used to collect the information. The administra- 
tor then coordinates the various program evaluation activities to have a total 
picture of operations within the center for decision making. One of the advan- 
tages of this arrangement is that evaluation may be seen as a team effort 
because a number of staff are involved in the process. This attitude may con- 
tribute to the acceptance of change in the organization as well. But, the 
administrator's other responsibilities may have a higher priority over pro- 
gram evaluation so that very little is done. Even in the smallest centers, 
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some kind of rudimentary program evaluation must be done to prepare reports 
required by funders and to monitor clinical activities. Often center admin- 
istrators do this kind of practical program evaluation without realizing it 
when they make judgments based on available data, observations of the cen- 
ter's operations and regular reports from business and clinical staff. 

Outside E valuat ion by Consultants 

Centers without evaluation capabilities may engage outside consultants 
to conduct program evaluation . There are some definite limitations to this 
alternative, particularly in regard to the use of evaluation for internal 
management. Some centers have had success in employing outside consultants 
to set up information systems, process data and compile the statistical 
reports that are required by funders. These program evaluation activities 
can be very useful to a center if the administrator receives regular peri- 
odic reports from the outside consultants to make decisions regarding the 
internal management of the organization. Many centers need outside consult- 
ants to assist them in determining the kinds of program evaluation methods 
that are most useful for the center in training staff in evaluation proce- 
dures. The program evaluation process may be only marginally useful, however 
as in the case of information systems, unless the administrator has access to 
these data for internal management. Outside consultants are helpful in 
identifying the weaknesses in management and clinical service delivery pro- 
cedures, but the administrators themselves, because of their working know- 
ledge of center operations, should make the decisions regarding changes to 
be made within the center. If at all possible, the administrator should 
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include key staff in the decision making process. Generally, contracting with 
outside consultants for evaluating centers in any other context is not satis- 
factory because the consultant usually lacks the intimate knowledge of the 
center's philosophy and operating problems and is thus inclined to make 
unrealistic recommendations for changes in procedures that may be costly and 
disruptive for the center. 

STAFFING 

Franklin and Thrasher state: "...ideally, evaluators should have exper- 
tise in statistics, cost accounting, management, public relations, anthro- 
pology, sociology, political science, psychology, philosophy and, of course, 
the content of the program to be evaluated. Given this impossible set of 
qualifications, it is no surprise that there are few 'fully qualified' eval- 
uators, although there are hundreds so labeled and so employed. This 
description suggests that an evaluator in a mental health center must have 
a broad range of skills which are usually acquired through experience in the 
field and are not representative of any particular academic discipline. 
Furthermore, many centers with minimal funds for program evaluation are unable 
to employ full-time evaluators. Others who may have the funds prefer to dis- 
perse evaluation responsibilities among varied staff members. Both of these 
arrangements are probably more representative of the organizations that are 
considered "average" in this publication. But, the person or persons who 
perform evaluation tasks need relatively similar skills and training. 

The administrators of many centers prefer to either employ individuals 
who have had previous experience in program evaluation. in state or local 
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mental health agencies or assign responsibilities for evaluation to one or 
more staff members already in the organization. Although disciplinary back- 
ground and degree level may be minor considerations, it appears that basic 
skills and competencies are major factors in choosing an evaluator. These 
basic skills and competencies include both technical skills and interper- 
sonal skills. It may not be necessary to have a doctoral or even a master's 
level evaluator. It depends on the evaluation tasks that are required. A 
bachelor's level evaluator may have the skills needed if the administrator 
can provide adequate supervision. 

Desirable Skills and Training 
Technical knowledge and skills: 

• program evaluation technology 

• mental health planning methodology 

• social systems analysis and some knowledge of 
experimental research 

• statistical analysis 

• understanding of computer technology (if computers 
are used) 

• understanding of the mental health delivery system and 
related government funding and reimbursement (third 
party payment) programs 

§ data management and use of information 

§ human service organizations and management procedures 

§ cost accounting 

• familiarity with clinical procedures 
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Interpersonal skills and orientation: 

• social skills and orientation to fit into the organiza- 
tion as a team member, not as an isolated individual 

• ability to organize work and meet time limits 

• motivational skills to encourage others to be involved 
in program evaluation and to use findings 

• group process skills to assist in bringing group con- 
sensus in decision making within the center 

• ability to abstract, conceptualize and recommend alter- 
natives based on interpretation of data and knowledge 
of center operations 

• commitment to the maintenance and growth of the 
organization 

This listing represents the ideal combination of skills and training not 
often found in a single individual. Some of these technical skills can be 
acqu'red through working in a center, attending workshops and reading, but 
most of the interpersonal skills and orientation appear to be a prerequisite 
to employment as an evaluator. 

The selection of a program evaluator depends largely on the priorities 
set by center management and the skills of present staff. One center admin- 
istrator may place emphasis on cost studies and be less concerned about the 
evaluator*s skills in clinically oriented studies, while another administra- 
tor who has an accountant with a strong background in cost studies may want 
an evaluator whose background in clinical procedures would complement those 
of the center's fiscal staff. When using the skills of the present staff to 
do program evaluation, the center administrator should tap the staff who 
have an interest in and grasp of evaluation procedures and encourage them to 
improve their skills. 
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The variation in organization and staffing of mental health centers 
makes the possible range of functions of the evaluator equally variable. 
Given the assumptions that the center administrator is primarily responsible 
for program evaluation and the person who serves as program evaluator is part 
of the management team, some functional relationships can be generalized . In 
the context of this publication, the management team includes the center 
director and the key supervisory staff responsible for both administrative 
support services and clinical services. Depending on the size and organiza- 
tional structure of the center, as few as two people and as many as six or 
eight may share these responsibilities. The program evaluator should be 
included in meetings of this group as a peer so that he may be sensitive to 
the goals of the organization (both formal and informal), the values and 
attitudes of management and the internal and external social, economic and 
political expediencies that influence decisions. 

In addition, the program evaluator should have a close working relation- 
ship with each supervisory level individual so that he knows their responsi- 
bilities and evaluation needs. The center administrator should support the 
evaluator in establishing these relationships, but the evaluator is primarily 
responsible for maintaining them through written and interpersonal 
communication. 

As part of the management team, the evaluator serves the center in many 
ways. Knowledge of management procedures is necessary because the evaluator 
must work closely with center management in a number of capacities and should 
be flexible enough in skills and attitudes to assume a broad range of 
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assignments that may, on the surface, appear to be unrelated to a narrow 
definition of program evaluation. The e valuator in a small center should be 
prepared to do routine evaluation tasks, not sophisticated research. For 
example, the evaluator may collect data and monitor the adequacy of records 
for completeness. These activities, though time-consuming and tedious, are 
worthwhile if used in preparing management oriented evaluation reports. In 
addition to preparing reports required by funding agencies, the evaluator 
will probably be responsible for monitoring statistical data on the center's 
operations. On some regular schedule the evaluator should be able to provide 
the administrator and others on the management team with program evaluation 
findings that refine observations and statistics regarding center operations 
so that the managers can make valid, rational decisions about changes in 
programs. Annual planning and proposal writing are other areas that the 
evaluator will be involved in as part of the management team. The evaluator 
will at least be expected to provide data for, and may be primarily responsi- 
ble for, both of these activities. 

The evaluator should have a working knowledge of clinical cedures and 
be able to work with the clinical director(s) and staff w' "hr antagonizing 
them, especially those who are resistant to evaluation. He or she may be 
responsible for monitoring clinical records for completeness and may be 
involved in quality assurance procedures. Also the evaluator should be 
responsive to requests by clinical directors for special studies regarding 
clients and programs. 
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In fiscal management areas, the evaluator may be assigned responsibility 
for organizing the center's information system so that data on client move- 
ment, staff activity and cost can be readily accessed when needed. He or 
she may also be involved in accounting activities as they relate to third 
party programs, funding, planning, cost analysis studies and proposal writing. 
The evaluator therefore should have an understanding of funding and reimburse- 
ment programs and basic accounting procedures. 

Under ordinary circumstances the evaluator should not be expected to 
relate directly to the governing boards and groups external to the organ- 
ization unless asked by the center administrator to serve as his represent- 
ative or to make program evaluation presentations to decision making groups. 
In these situations, the evaluator must avoid formal scientific affectations 
and relate to these groups as a representative of the organization who is 
committed to its maintenance and growth and who communicates in a way that 
is easily understood by laymen. 
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Chapter IV 

ADMINISTRATIVE USES OF PROGRAM EVALUATION 



The preceding chapters have discussed program evaluation functions and 
alternative ways to organize and staff for these functions. Although "the 
evaluator" in the organization has been referred to, it is recognized that 
many center administrators may not be able to employ a full-time evaluator 
but instead must assign evaluation responsibilities to dual purpose units, 
such as program evaluation and planning, or to individual staff members. By 
assigning program evaluation responsibilities to varied staff members per- 
forming other management and clinical functions, the administrator may in fact 
encourage the acceptance of program evaluation because of group participation 
in the process. 

CREATING A POSITIVE CLIMATE FOR PROGRAM EVALUATION 

Whatever the organization and staffing plan, the key to successful 
incorporation of program evaluation in the center is the administrator's 
leadership and management skills in creating a positive climate for program 
evaluation and making those responsible for program evaluation a part of the 
management team. Without the full support and involvement of the adminis- 
trator, the program evaluator may simply provide the required reports to 
funders but not improve the management process. This kind of activity may 
be met with resistance by staff because they receive no feedback on their 
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reporting. When administrators decide to introduce program evaluation in 
centers, they should not lose sight of the purpose of mental health centers — 
to provide the best possible care to the community with the resources avail- 
able. Two very basic goals are involved: the need to improve management pro- 
cedures and the need to maintain or improve the quality of care provided to 
clients. Both goals require cooperation within and between different organ- 
izational levels in the center. For example, a center may have excellent 
management control for all aspects of the center's operations but the quality 
of care may be less than desirable because the management system does not 
allow the flexibility needed to provide appropriate treatment for a wide 
range of clients with differing needs. As a result, the morale of clinicians 
may be low and their resistance to regimentation and "red tape" high. Another 
center may provide a high quality of care to clients and have no service 
delivery problems but may not survive in the long run because of poor manage- 
ment. Obviously, the center administrator must seek some balance between 
these two goals in order to serve the purposes for which the mental health 
center was established. The introduction of center-wide program evaluation 
can set che stage for both improving the quality of care and providing sound 
management practices by offering systematic data for decision making about 
organizational change. 

Planning and Implementation 

When beginning program evaluation in a mental health center, the center 
administrator should introduce the new procedures in as positive a way as 
possible. This can start by bringing together a few key staff members who 
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will be involved in program evaluation to plan and design a set of program 
evaluation methods and procedures. If both the director and the staff have 
only limited knowledge of program evaluation, the administrator may arrange 
for an outside consultant to assist in designing the organizational structure 
and ongoing procedures for program evaluation, and in identifying the best 
ways for setting goals and objectives using available resources. The result 
of these planning sessions should be a working plan for implementing program 
evaluation in the center. 

During the planning and implementation process it is important that the 
administrator, the evaluator and/or key staff keep other staff informed of 
their activities and seek participation and involvement whenever appropriate. 
Staff conferences may be held to clarify: a) the purpose of program evalua- 
tion procedures (e.g., if beginning program evaluation and better management 
of resources are necessary for the economic survival of the center, an 
explicit statement to this effect should be made to staff); b) the coopera- 
tion expected of individual staff members (e.g., filling out and submitting 
forms, negotiating realistic objectives and action steps); and c) the varied 
uses of evaluation data (e.g., required reporting, planning, monitoring and 
summary reporting with feedback to staff, outcome evaluation, special studies 
identifying needed changes in program operation). These staff orientation 
sessions should be presented in a way that emphasizes the positive uses of 
program evaluation and demonstrates the full support of the administrator and 
the management team and the sensitivity of the program evaluator to feedback 
from staff or to any requests for information made by staff. 
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It is suggested that the administrator consider including several man- 
agerial procedures for program evaluation. 

• Clarify procedures for how various kinds of data are to be used . 
Some clinical information should only be available to certain 
staff (e.g., program director, clinicians). Othsr information 
may be suitable for general release (e.g., mass media, community 
groups, local government and others). 

• Set time limits for the program evaluator's activities , par- 
ticularly for required reports to funders and summaries of 
monitored data. Both the administrator and the evaluator 
benefit from this arrangement because the evaluator can sched- 
ule the work according to priorities and plan a format for 
data collection and tabulation that will allow these data to 
serve multiple purposes. Because staff turnover is not uncom- 
mon in mental health centers, it is recommended that the work- 
ing plan for program evaluation, the time schedules and the 
action steps be written so that replacement staff can carry 

on the evaluation activities if necessary. 

There are other recommendations to be considered regarding the role of 
the evaluator. 

• The evaluator should not be left alone to decide what to eval- 
uate . Instead, the administrator should let the evaluator know 
what kind of information is needed for decision making and 
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for communication with groups outside the center. The evaluator 
is then able to decide what kind of data must be collected to 
assist the administrator in making these decisions. 

• T he administrator and the evaluator should consult with pro - 
gram directors to determine their needs for program evaluation 
information for management uses. This procedure encourages 
program directors to cooperate with the evaluator and use 
evaluation findings constructively. 

• The evaluator should not be a detached researcher operating 
in isolation within the center . The evaluator needs to be in 
the mainstream of center activities in order to anticipate 
needs for data and make recommendations for alternative ways 
of resolving problems within the center. However, the admin- 
istrator is still responsible for integrating program eval- 
uation activities into the center and encouraging the use of 
findings, 

• Staff and operating expenses for program evaluation activit i e s 
should be identified and allocated in the center's budget . 
accounting procedures (cost centers) and accountability procedures . 

• The evaluator must develop sources of information through what 
appear to be diversionary activities (e.g., organized clinical 
records, setting up an integrated information system, writing 
proposals, etc.) before he can begin to evaluate programs. If 
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a center n just beginning formal program evaluation services, 
the evaluator may spend considerable time doing tasks that 
seem unrelated to program evaluation. An evaluator does not 
usually join an organization and immediately start evaluating.^ 

USING PROGRAM EVALUATION AS A MANAGEMENT TOOL 

Chapter II outlines the three stages of program evaluation functions in 
a mental health center: 1) planning; 2) monitoring; and 3) assessment. The 
information provided by these program evaluation functions should be used not 
only for preparation of reports required by funders but as management tools 
to improve center management in three areas: 1) planning and development of 
programs; 2) managing of ongoing programs; and 3) reporting to governing 
boards and to other community organizations. 

Planning for Programs 

Data on staff activity, client movement, costs and basic demographic 
characteristics of clients collected by information systems can be used to 
conduct many of the program evaluation activities described in Chapter II. 
These results are useful to management for program planning and developing 
new programs . Special studies may be needed to provide planning inputs that 
are not available in a center's information system. 

Many mental health center administrators are shifting to " management by 
objectives" to plan for center operations . This approach involves setting 
goals and specific time-defined objectives with criteria measures to deter- 
mine the degree of attainment of these goals and objectives . These goals 
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and objectives may be set annually for the center as a whole, for selected 
organizational levels within a center (e.g., governing board, administrative 
support services, clinical services), for specific programs and for individual 
staff performance. Specific goals and objective's may range from serving more 
clients of a certain kind to increasing the hours ^ service provided by an 
individual clinician. Basic criteria for these goals and objectives are 
described in Chapter II. 

Operating under "management by objectives" can create the needed balance 
between good management practices and the delivery of quality of care if 
realistic behavioral objectives and specific action steps are negotiated 
between management and staff. If adhered to and used positively, "management 
by objectives" defines objectives that are clearly understood, accepted by 
and attained by both staff and management. In setting goals and objectives 
for the coming year, the administrator focuses on what has been done in the 
past and states what the expectations of the organization and its staff will 
be for the next year of operation. This approach assists the administrator 
and other supervisory staff in managing programs and also provides the e val- 
uator with criteria for evaluating program outcomes. 

• Administrators may have difficulties beginning "management 
by objectives" procedures if they do not have adequate data 
describing the baseline performance and criteria measures. 
In such an instance, the administrator and key staff may set 
initial objectives that seem reasonable and refine them 
quantitatively as baseline data becomes available. 
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• Administrators may meet with resistance to the "management 
by objectives" approach because some clinicians find setting 
concrete objectives contrary to their philosophies, while 
other staff do not want to commit themselves to clearly 
defined objectives which would limit their flexibility. 
These difficulties may be resolved through strong leader- 
ship by the governing board, the administrator and manage- 
ment level staff and by negotiating with clinical staff in 
order to develop objectives that define the center's expec- 
tations but at the same time allow clinicians enough flexi- 
bility to perform tasks without unrealistic constraints. 

§ The negotiation of objectives with staff may pinpoint areas 
where inservice training or continuing education in mana- 
gerial and clinical skills are needed. If individualized 
objectives are negotiated, the development of appropriate 
skills may be included in the objectives set for individual 
staff members. 

§ The results of assessments of the needs and expectations for 
programs are useful in both setting goals and measuring pro- 
gram outcomes. Often center management overlooks the potential 
uses of these studies to determine whether a new program is 
actually providing services to target population and whether 
the expectations for services are being met. 
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Managing Programs 

Evaluation data drawn from monitoring reports are used in the day-uo-day 
management of programs . The manager uses the monitoring reports like a ther- 
mostat to show when activities going on in a center do not meet an accepted 
or expected range. These data include client movement, staff activity, staff 
productivity and costs of services. Other data can be monitored for compli- 
ance with standards set by administrative agencies or national accreditation 
organizations. The primary management use of monitoring is to idenfity needs 
for immediate corrective action in ongoing programs. 

Data on client movement, staff activity, staff productivity and cost of 
services should be continuously monitored and compiled in periodic summary 
reports for use by the administrator, fiscal managers, and program directors 
in managing programs. When the data findings in any of these areas reflect 
significant changes (e.g., underuse or overuse in relation to present goals 
and objectives), corrective action should be taken by management. Such data 
may also be used to provide reports to state and federal agencies and to pro- 
vide the baseline data needed for planning and goal setting. (Additional 
uses include tracing referral patterns and demographic characteristics of 
clients, determining patient flow through the center's network of services, 
identifying treatment patterns for various groups of clients and providing 
information needed to reallocate resources in the annual planning process. 2} 
Centers which administer client outcome measures may also monitor these out- 
comes for changes. 
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pMonitoring for compliance with standards yields information that the 
administrator can use to identify problems in treatment and utilization pat- 
terns which require corrective action. Staffing standards can be used to 
identify manpower needs and to plan for staff training. The administrator 
may also conduct surveys of the center's facilities to discover deficiencies 
in building and equipment standards before site visits from state and federal 
administrative agencies or national accreditation organizations. 

When the data from monitoring reports do not adequately answer questions 
about the program, the center administrator may ask the evaluator to conduct 
special studies to determine what action should be taken. This "red flag- 
ging" or "evaluation by exception" provides for in-depth answers to questions 
raised by the monitoring process. 

Some of the problems related to program monitoring are: 

• Many centers do not yet have an organized information system 
In operation. Data on client movement, staff activity and 
costs are collected, but they are not organized for monitor- 
ing purposes and may not report on all aspects of center 
operations (e.g., many centers do not report on 100 percent 
of staff time). 

• Data on staff activity and productivity and client movement 
(measures of effort) are often not appropriately correlated 
with cost figures to identify the actual cost of services 
(cost accounting). 
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• Many centers have difficulty deciding on appropriate units of 
reporting because various state and federal funding agencies 
use different units and categories in their required statistical 
reports. This is a broad problem related to the requirements 

of administrative agencies for different forms of the same data. 
Although not directly related to the internal management of 
centers, it should be noted. 

• Many centers delay in beginning any information system and 
monitoring procedures because they cannot yet afford an auto- 
mated system. However, a carefully designed manually operated 
system can adequately capture the basic data needed to monitor 
and assess mental health program activities. 

Uses of Evaluation Findings 

Routine program evaluation activities vary from center to center. One 
center may only evaluate staff activity, client movement and program costs 
on a regular basis, while another center may routinely evaluate client out- 
comes, cost outcomes and client satisfaction in addition to these basic eval- 
uation activities. Recognizing that this variation in evaluation activities 
will exist, we offer the following possible uses of program evaluation 
activities. 

Evaluation findings regarding staff activity and client utilization show 
the amount of effort used to provide services to a given number of clients. 
This information is helpful in estimating the amount of staff time needed in 
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programs. It also is helpful in determining whether specific client groups 
are actually using services, and whether procedures for the transfer of 
clients from one service or program to another are being completed in an ade- 
quate way. Information regarding the use of services and movement between 
services assists the center administrator in planning for changes in intake, 
transfer and follow-up procedures, location of services, referral relation- 
ships with other agencies, etc. Many of these changes can be written as 
objectives in the annual goal setting process if the resources for making 
these changes are available (e.g., reducing hospitalization of clients by 
increasing staff assigned to follow-up care for clients leaving state 
hospitals) . 

Cost studies are useful to the administrator in preparing budgets, set- 
ting program priorities, and making program changes. There are two basic 
procedures that are within the capabilities of most centers: 1) cost 
accounting which determines the actual cost of ongoing services and 2) 
cost analyses which provide estimates of the amount of resources that must 
be allocated to meet various program objectives. 

Cost outcome studies , which measure the cost of an episode of client care 
in relation to the actual client outcome (e.g., improved social functioning 
after treatment) can also be done if data on both client outcome and cost per 
episode are available. Cost outcome findings can be particularly useful in 
demonstrating the overall effectiveness of programs to funders and other 
interest groups. 
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v^uMiparaLive cost eTTGcti veness and cost benefit studies are probably 
beyond the scope of many centers for the reasons discussed in Chapter II. 

Client outcome measures are usually administered to clients at least 
twice (pre- and post-treatment) by clinicians in order to measure overall out- 
comes of services (e.g.. goal attainment scaling and standardized scales). 
These client measures are often useful to clinicians for developing individ- 
ual treatment plans. Some programs or services may administer these outcome 
measures to all clients, then aggregate scores to measure overall program out- 
come. When they are available, program outcome findings can be used to deter- 
"1^""^ impact of services on clients as well. Further, it is possible to 
measure cost outcome when program findings are merged with data on the costs 
of care. The results of service outcomes can assist the administrator in 
determining whether current treatment models are providing the best possible 
care in relation to the cost involved, and in reallocating resources to var- 
ious services. Comparisons of outcomes between various services can be made 
with outcome data; however, these comparisons generally require more sophis- 
ticated research design and controls than may be feasible in many centers. 

Special p rogram evaluation studies are usually undertaken when further 
refinement of existing data (e.g., routine information system data, clinical 
records, billing and accounting records) is necessary or when questions can- 
not be answered by the available reports. Occasionally the administrator or 
the evaluator will see an unusual change in monitoring data which requires a 
special study. Centers with limited staff conduct special evaluation studies 
only when there is a clear need for findings. The evaluator should prepare 
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the findings of these studies in a form that is understandable to the users 
of this information. It is suggested that the evaluator submit preliminary 
findings to the administrator while the final results are being completed, 
and that the evaluator discuss the preliminary findings with people who are 
directly involved (e.g., the administrator, directors of programs) to ask 
for their reactions to the data and their interpretations while the data are 
still in the preliminary stages. When preparing the final report, the eval- 
uator should submit alternative recommendations for changes in a program based 
on the suggestions made by others. It must be emphasized, however, that it 
is the responsibility of the center administrator, not the evaluator, to 
assume responsibility for making program changes. 

A "no show" study is an example of a simple special study that may be 
done for the outpatient service. The evaluator collects and aggregates data 
on the number of "no show" clients and notes the times when clients are more 
inclined to use services. The administrator may then wish to reschedule 
staff meetings and other administrative activities to hours when clients are 
less apt to keep appointments or to change the hours that services are avail- 
able (e.g., evening hours). 

A special study may be done at a time of crisis when a center or pro- 
gram is under the scrutiny of groups in the community. In such instances, 
the evaluator will be called upon to draw from existing data in the center 
and to identify additional information needed so that the administrator can 
reply to the outside requests. 
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Reporting to Others 

The center administrator can use evaluation findings to assist the cen- 
ter's governing board in making decisions and to report to community groups 
on the center's progress and problems . 

Governing boards are responsible for planning and setting of policy for 
a center's overall operations. In order to make rational decisions, board 
members should have accurate information about the political, economic and 
social climate in which the center operates, the number and kinds of people 
who need mental health-related care, the actual services provided to these 
people by the center, its existing organizational structure and staffing 
patterns and the costs of operating the center and its programs. Program 
evaluation answers some of these questions, but social values and political 
processes sometimes modify the course of action that rather rigorous program 
evaluation findings would suggest. Nevertheless program evaluation is 
extremely useful in assisting the administrator to have well developed find- 
ings for the decisions that need to be made by a governing board. Program 
evaluation also presents other explanatory information which is useful in 
educating board members so that they have the understanding needed to make 
their decisions on operations and policy. The following examples show how 
program evaluation can be used for both regulatory and educational functions. 

Monthly or quarterly status reports on programs are distributed at board 
meetings along with a brief written or oral analysis and explanation. Fre- 
quently it seems that board members' primary interest in the fiscal 
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operations of a center. However, in providing information on program oper- 
ations, the administrator may expand the board^s participation and understand- 
ing of the programmatic issues and problems facing the center. 

Monitoring for compliance with standards discussed in Chapter II is 
another activity in which the governing board may be involved. Board members 
can be included in the pre-site visit survey of the center to encourage their 
understanding of the requirements of administrative and accrediting agencies. 

Although a governing board is responsible for major planning and policy 
setting for a center, both the board and the center management have a respon- 
sibility to be accountable to the broader community. 

Accountability as a general term means that a person is respon- 
sible to someone else for accomplishing certain results with the 
resources available to him. In the mental health field, program 
accountability refers to the responsibility of tht agency or pro- 
gram staff to produce certain kinds of results with the funds and 
other resources allocated to it.^ 

There is a wide range of individuals and groups in the community to 
which a center may be accountable. Some of these groups are: 

• Users of mental health services : individual clients 

(including consultation and senior citizen associations 

education) schools 

youth organizations 

law enforcement agencies 

nursing homes 
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• Groups who refer clients : 



human service organizations 

medical doctors 

clergy 

courts 

law enforcement agencies 

schools 

nursing homes 



t Loca'i funders: 



city and county commissions 
community agencies 



• Public interest groups : 



citizen advisory groups 
mental health associations 
service organizations 
health councils 
neighborhood associations 



• Planning agencies : 



regional planning councils 
health systems agencies 



The center administrator can use program evaluation Information to 
explain to these groups what the center is doing, how much services cost and 
why the services are valuable to the community and should be supported. This 
Information may be communicated in written reports, at public meetings, pre- 
sentations to community groups and through the news media, depending on the 
audience and the issues Involved. There are two kinds of program evaluation 
reports: regular, periodic reports and reports that respond to particular 



Regular, periodic reports on center activities may Include: 

§ annual reports of the mental health center distributed to 
various groups In the community; 

• reports made to local funding agencies at regularly scheduled 
budget sessions; 

• reports to citizen advisory groups and other public interest 



Issues. 



groups. 
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These reports are often fiscally oriented, because they describe how 
the clients are served with the funds allocated to the center. But when addi 
tional information about the nature and impact of services on clients and 
the formal linkages with other groups that refer clients or provide ancil- 
lary services to clients is included, these reports can be very helpful in 
demonstrating the value of mental health services to the community and in 
winning support. 

Periodic newsletters are also an effective way of communicating infor- 
mation about center activities, particularly when evaluation reports are 
included with information showing the value of a particular program or serv- 
ice to the community. For example, a newsletter may describe how services 
have helped a specific subgroup of clients and show how many of these people 
have been served with the available funds. 

Reports that respond to a particular issue vary considerably in purpose 
and content. They are often directed toward different groups with different 
interests. Sometimes these reports require a special evaluation study to pro 
vide documentation of a specific aspect of the center *s operations. The eval 
uation data must be translated into concise narratives, aggregate statistics 
and simple graphic displays that present the information in an easy-to-under- 
stand, accurate way to an audience that is relatively unfamiliar with the 
structure and operation of the mental health delivery system. Some of the 
ways that evaluation reports may be used are: 
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• supporting documentation for requests to local funding groups 
for increased allocations to maintain existing services or to 
begin new ones; 

• enhancing the community's awareness of ongoing services or the 
need for new services for a particular subgroup of clients; 

• encouraging referral linkages, regular consultations and 
educational programs for users of mental health services or 
groups who refer clients; 

• providing information that will assist planning agencies in 
planning for and reviewing mental health program applications; 

• responding to crisis situations when the center is under 
criticism from groups in the community. 

Reports to the corrmunity can have a strong influence in informing inter- 
ested groups and individuals in the community about a center's activities 
and can be helpful in gaining the support of these groups. The center ^s 
board and administrator should be sensitive to feedback from these groups, 
using their recommendations to provide clearer documentation of the expec- 
tations for mental health services in the community. Also, by encouraging 
feedback from these groups, the board and the administrator may discover 
potential resources available in the community, such as contract funds, vol- 
unteer workers, and in- kind contributions and services that will assist in 
providing mental health rare to the community. 
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